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ABSTRACT 


This  document  has  been  prepared  by  Jeff  C.  Gust  (Quametec  Corp.,  under  contract)  in 
collaboration  with  NIST  Weights  and  Measures  Division  (WMD)  to  enable  State 
Weights  and  Measures  Laboratories  to  comply  with  criteria  for  proficiency  testing  as 
noted  in  NIST  Handbook  143  and  as  needed  for  accreditation  programs.  This  document 
is  intended  to  provide  the  basic  quality  system  for  Proficiency  Testing  and  Interlaboratory 
Comparisons  developed  by  NIST  Weights  and  Measures  Division. 
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INTRODUCTION 

Scope 

This  document  has  been  developed  for  the  NIST  Technology  Services,  Weights  and 
Measures  Division  (WMD)  in  order  to  establish  and  implement  a quality  system  for  the 
planning,  implementation,  analysis,  and  reporting  of  Proficiency  Tests  (PT)  and 
Interlaboratory  Comparisons  (ILC)  conducted  within  the  WMD  Measurement  Assurance 
Program  (MAP).  This  document  contains  the  NIST  QMD  Quality  Manual  for 
Proficiency  Testing  and  Interlaboratory  Comparisons.  This  document  provides  the  quality 
system  to  ensure  that  all  Proficiency  Testing  and  Interlaboratory  Comparison  activities 
within  WMD  are  compliant  with  ILAC-G  13:2000,  Guidelines  for  the  Requirements  for 
the  Competence  of  Providers  of  Proficiency  Testing  Schemes  and  ISO/IEC  Guide  43- 
1:1997,  Proficiency  Testing  by  Interlaboratory  Comparisons  - Part  1,  Development  and 
Operation  of  Proficiency  Testing  Schemes.  This  document  specifies  requirements  to 
ensure  that  NIST  WMD  and  associated  delegates  are  technically  competent  to  provide 
specific  types  of  proficiency  testing  schemes  as  required  by  NISTIR  7082,  Proficiency 
Test  Policy  and  Plan  (for  State  Weights  and  Measures  Laboratories),  2004. 

Document  Retention 

This  document  has  no  expiration  date  and  remains  in  effect  until  revised,  replaced,  or 
retired. 

References 

Compilation  of  Federal  Ethics  Laws  (approved  through  May  27,  2003)  - Prepared  by  the 
United  States  Office  of  Government  Ethics.  [Web  page], 
http://www.usoge.gov/pages/laws  regs  fedreg  stats/comp  fed  ethics  laws.pdf 

Gust,  J.  C.  and  Harris,  Georgia  L.  Proficiency  Test  Policy  and  Plan  (for  State  Weights 
and  Measures  Laboratories).  National  Institute  of  Standards  and  Technology,  (U.S.) 
NISTIR  7082,  2004  January  [Web  page]  http://www.nist.iJov/labmetroloizv. 

Harris,  Georgia.  L.  State  Weights  and  Measures  Laboratories  Program  Handbook,  4th 
Edition.  National  Institute  of  Standards  and  Technology  (U.S.)  Handbook  143;  2003, 
March.  [Web  page]  http://www. nist.gov/labmetroloizv. 

Harris,  Georgia.  L.  State  Weights  and  Measures  Laboratories:  State  Standards  Program 
Description  and  Directory.  National  Institute  of  Standards  and  Technology  (U.S.)  NIST 
SP  791,  2000  Edition;  2000  August.  [Web  page], 

http://ts.nist.gov/ts/htdocs/230/235/labdirectorv/sp79 1 .htm  [last  updated  10/1/02], 
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Definitions 

Analyst  - Assigned  person  who  is  responsible  for  handling  all  data  related  activities  for 
the  Proficiency  Test  or  Interlaboratory  Comparison  (PT/ILC).  (As  used  in  this  publication 
“PT/ILC"  is  used  generically  to  mean  either  a Proficiency  Test  or  an  Interlaboratory 
Comparison;  the  specific  objectives  and  context  will  determine  the  application  and  use  of 
PT/ILC  results.) 

Coordinator  - A party  who  is  responsible  for  implementation  of  the  PT/ILC. 

Delegate  - Any  party  outside  of  NIST,  Technology  Services,  WMD  who  volunteers  or 
accepts  a task  related  to  the  PT/ILC  from  the  Technical  Manager  who  works  with  the 
WMD  to  accomplish  its  proficiency  testing  objectives,  whether  volunteer  or  contractor. 
Delegates  may  include:  analysts,  coordinators,  pivot  laboratories,  reference  laboratories, 
and  regional  administrators. 

Interlaboratory  Comparison  - a generic  term  for  the  circulation  of  an  artifact,  standard,  or 
set  of  standards  for  the  purpose  of  evaluating  measurement  results. 

NIST  POC  - Assigned  point  of  contact  within  NIST  WMD  for  the  PT/ILC. 

Participant  - Any  organization  meeting  RMAP  (see  definition)  requirements  for 
participation  in  a PT/ILC. 

Pivot  Laboratory  - Laboratory  organization  that  determines  stability  value  for  the  artifact. 

Provider  - NIST  Weights  and  Measures  Division  is  considered  the  “provider.*' 

Proficiency  Test  - A specific  interlaboratory  comparison  that  may  be  used  by  a 
laboratory  or  Recognition  and/or  Accreditation  bodies  as  evidence  of  staff  competence. 

Reference  Laboratory  - Laboratory  organization  that  determines  the  reference  value  for 
the  artifact. 

RMAP  - Regional  Measurement  Assurance  Program. 

Regional  Administrator  - Regional  coordinator  who  assists  in  tracking  the  status  of  all 
PT/ILCs  within  the  region. 

Technical  Manager  - The  Technical  Manager  is  the  Group  Leader  for  the  WMD 
Laboratory  Metrology  Group. 
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1.  ADMINISTRATION 

1.1.  Quality  System  & Policy 

1.1.1.  This  document  establishes,  implements  and  maintains  the  quality 
system  for  proficiency  testing  and  interlaboratory  comparison  activities  of 
NIST  WMD  for  the  scope  of  Measurements  that  are  defined  in  section  1 of 
NISTIR  7082,  Proficiency  Test  Policy  and  Plan  (for  State  Weights  and 
Measures  Laboratories). 

1.1.2.  This  document  defines  and  documents  the  WMD's  policy, 
objectives  and  commitment  to  ensuring  and  maintaining  quality  of  all 
aspects  of  proficiency  testing.  Artifact  quality,  characterization,  and 
assignment  of  property  values  are  described  in  PLANNING.  Evaluation  of 
each  laboratory's  performance  and  statistical  treatment  of  test  results  are 
described  in  ANALYSIS.  Distribution  of  artifacts,  storage  and  transport 
procedures  are  described  in  IMPLEMENTATION . Reporting  procedures 
are  described  in  REPORTING. 

1.1.3.  Quality  Policy:  NIST  WMD  pledges  to  provide  proficiency  testing 
services  using  metrologically  sound  practices  and  the  highest  levels  of 
service  quality  as  documented  in  this  quality  system.  All  staff  and 
delegates  associated  with  proficiency  testing  services  pledge  to  adhere  to 
the  policies  and  procedures  documented  in  this  quality  system,  performed 
in  accordance  with  ILAC  G 13:2000,  1SO/IEC  Guide  43-1,  and  relevant 
sections  of  ISO/IEC  17025:1999. 
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1.1.4.  The  locations  of  the  applicable  sections  of  ILAC  G13  are  as 

follows: 


ILACG13  1 

Requirement,  Section  2.1.3,  section  letter 

Quality  Manual  Section 

a 

Aims 

Introduction 

a 

Scope 

NISTIR  7082 

a 

Statistical  Design 

Analysis 

a 

Format 

Reporting 

b 

Operational  Procedures 

Implementation 

c 

Preparation  and  Issuing  of  Reports 

Reporting 

d 

Policies  on  Confidential  and  Ethical  Procedures 

Administration 

e 

Computing  and  Information  Systems 

Administration 

f 

Collaboration  and  Sub-contracting 

Administration 

g 

Fees  for  Participation 

Administration 

h 

Scope  of  Availability  of  Proficiency  Testing 
Programs 

Regional  MAP  Programs 

i 

General  Policies  on  Participation 

Administration 

J 

Use  of  Scheme  Results 

Administration 

k 

Procedures  for  Handling  Complaints 

Administration 

1.1.5.  Throughout  this  document,  the  responsibilities  for  all  activities  are 

defined  in  each  applicable  section. 

1.2.  Organization  and  Management 

1.2.1.  The  organization  providing  proficiency  tests  within  the  scope  of 
this  document  is  NIST,  Technology  Services,  Weights  and  Measures 
Division  (Division  260.) 

1.2.2.  The  Technical  Manager  and  Quality  Manger  (as  assigned  by  the 
WMD  Laboratory  Metrology  Group  Leader)  task  other  personnel  within 
the  WMD  Metrology  Group  and  Delegates  to  complete  all  aspects  of 
PT/ILC  within  the  scope  of  this  program. 

1.2.3.  In  order  to  ensure  that  all  management  and  personnel  are  free  from 
any  commercial,  financial,  internal,  or  external  pressures  that  may 
adversely  affect  the  quality  of  PT/ILC  activities,  all  Federal  Employees  are 
subject  to  abide  by  the  Federal  ethics  laws  as  cited  in  the  “Compilation  of 
Federal  Ethics  Laws”.  Additionally,  all  appointed  Delegates  must  comply 
with  all  elements  contained  in  POLICY  FOR  PARTICIPATION  which  is 
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intended  to  ensure  freedom  from  external  pressures.  These  policies  also 
ensure  that  all  personnel  involved  in  PT/ILC  activity  avoid  involvement  in 
activities  that  might  diminish  the  confidence  in  the  competence, 
impartiality,  judgment,  or  operational  integrity  of  NIST  WMD. 

1.2.4.  It  is  the  policy  of  NIST  WMD  that  all  PT/ILC  activities  are 
completely  open,  with  no  coded  or  confidential  information.  All 
participants  must  agree  to  these  conditions  before  being  qualified  to 
participate  in  PT/ILC  activities.  This  document  may  be  used  to  develop 
PT/ILC  activities  for  the  NIST  National  Voluntary  Laboratory 
Accreditation  Program  (NVLAP)  at  their  request.  For  this  special  case, 
access  to  information  which  results  from  PT/ILC  activities  is  restricted  to  a 
minimum  number  of  people  and  the  reported  information  may  be  coded  to 
ensure  the  confidentiality  of  the  participants  where  NVLAP  determines 
that  it  is  necessary. 

1.2.5.  WMD  may  occasionally  coordinate  PT/ILC  activities  for  NVLAP 
or  other  accreditation  bodies  when  it  is  in  the  interest  of  WMD  program 
objectives.  In  these  cases,  WMD  will  follow  the  confidentiality  policies  of 
the  accreditation  bodies  if  requested. 

1.2.6.  NIST  WMD  has  developed  organization  charts  to  define  its 
Organization  and  Management  structure,  its  place  within  NIST,  and  the 
relations  between  management,  technical  operations,  support  services,  and 
delegates.  The  organization  charts  also  demonstrate  the  responsibility, 
authority,  and  interrelationships  of  all  personnel  who  manage,  perform  or 
verify  work  in  PT/ILC  activities. 

1.3.  Roles  and  Responsibilities 

1.3.1.  The  Technical  Manager  has  overall  technical  responsibility  for 
each  PT/ILC  activity.  The  Technical  Manager  or  Designee  from  NIST 
WMD  reviews  and  provides  final  approval  on  all  PT  data,  analysis,  and 
reports.  Final  approval  of  PT  data,  analysis,  and  reports  may  not  be 
assigned  to  personnel  outside  of  NIST  WMD. 

1.3.2.  The  Technical  Manager  must  have  experience  with  PT/ILC 
activities  that  are  conducted  within  the  scope  of  NISTIR  7082.  The 
Technical  Manager  must  have  the  educational  and  technical  background 
required  to  meet  a position  for  a Physical  Scientist  ZP  III  within  NIST 
WMD,  Laboratory  Metrology  Group.  The  requisite  background  includes 
extensive  experience  in  calibration  of  standards  of  mass,  length,  and 
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volume;  experience  in  developing  calibration  procedures  and  methods; 
participation  in,  or  coordination  of,  round  robins  or  other  interlaboratory 
experiments  and  comparisons;  and  experience  in  quality  system  operations 
and  audits. 

1.3.3.  The  Quality  Manager  has  responsibility  for  all  quality  related 
aspects  of  this  program  such  as  the  maintenance  of  records,  Internal 
Audits,  and  ensuring  adherence  to  policies  and  procedures. 

1.3.4.  The  Quality  Manager  must  have  experience  and  training  as  a lead 
auditor  in  a metrology-related  field  (e.g.,  ISO  9000,  ISO/IEC  17025). 

1.3.5.  The  Technical  and  Quality  Managers,  and  their  deputies  shall  be 
assigned  by  the  WMD  Laboratory  Metrology  Group  Leader  via  email  or 
memo,  which  are  retained  as  a correspondence  quality  record. 

1.4.  Collaboration  and  Subcontracting 

1.4.1.  The  Technical  Manager  may  designate  activities  from  the 
PLANNING,  ANALYSIS,  IMPLEMENTATION,  and  REPORTING  sections 
of  this  document  to  collaborators  who  will  hereby  be  known  as  Delegates. 

1.4.2.  Delegates  are  selected  based  upon  an  assessment  of  technical 
qualifications  as  determined  by  the  Technical  Manager  or  Designee.  All 
Delegates  are  evaluated  using  procedures  and  records  described  in  the 
Training  section  of  the  ADMINISTRATION  section. 

1.4.3.  Laboratories  that  are  selected  for  establishment  of  the  reference 
value  or  performance  of  pivot  measurements  must  be  either  accredited  in 
the  measurement  parameter  of  concern  by  an  accreditation  body  that  is  a 
signatory  to  the  ILAC,  MRA  (International  Laboratory  Accreditation 
Cooperation,  Mutual  Recognition  Arrangement  http://www.ilac.org),  or 
recognized  for  compliance  to  NIST  Handbook  143  by  NIST  for  the 
measurement  parameter  of  the  PT/ILC,  and  be  able  to  provide  a suitable 
level  of  measurement  uncertainty  for  the  proposed  PT/ILC.  Records  of 
competency  for  any  laboratory  used  for  measurement  activities  are 
maintained  in  the  WMD  files  and  databases.  Laboratory  qualification  may 
include  one  or  more  measurement  areas  and/or  PT/ILCs.  Laboratory 
recognition/accreditation  status  and  records  must  be  evaluated  for  accuracy 
and  updated  as  needed,  prior  to  each  assigned  PT/ILC. 
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1.4.4.  Subcontracting,  if  coordinated,  will  be  performed  with 

organizations  that  are  accredited  to  perform  PT/ILCs  and/or  comply  with 
this  quality  system. 


1.5.  Fees 

1.5.1.  NIST  does  not  charge  fees  for  participation  in  any  PT/ILC 
activities.  Commercial  Laboratories  that  participate  in  PT/ILC  activities 
are  to  pay  costs  of  shipping  to  and  from  their  laboratory.  This  requirement 
may  be  waived  if  they  are  serving  as  a delegate  for  the  PT/ILC. 

NOTE:  Fees  charged  for  attendance  at  the  RMAP  meetings  are  for  coordination 
and  costs  associated  with  the  meeting  functions  and  not  those  of  operating  the 
proficiency  testing  programs.  Attendance  at  the  meetings,  which  is  required  to 
obtain  reports  and  participation  in  the  discussions,  will  usually  require  payment 
of  fees  associated  with  the  cost  of  the  meetings  (distinct  from  the  cost  of  the 
PT/ILC  activity). 

1.6.  Policy  for  Participation 

1.6.1.  Any  laboratory  organization  (Participant)  that  would  like  to 
participate  in  a PT/ILC  activity  within  the  scope  of  this  document  must 
agree  to  operate  in  accordance  with  the  policies  and  conditions  for 
participation  as  described  in  this  quality  system  as  follows: 

1.6. 1.1.  The  Participant  will  follow  instructions  for  each  PT  and  use 
designated  procedures  when  instructed. 

1.6. 1.2.  The  participant  must  acknowledge  that  all  NIST  WMD  PT/ILCs  are 
considered  OPEN  and  that  anonymity  is  not  implied  or  guaranteed. 

1.6. 1.3.  The  Participant  will  not  use  any  PT/ILC  report  for  any  purpose  other 
than  internal  measurement  assurance  or  accreditation/recognition 
activities  (no  sales,  marketing,  or  advertising  of  the  results  of  any 
participating  laboratory). 

1 .6.1.4.  The  Participant  will  not  falsify  any  results  submitted  to  NIST  WMD. 

1.6. 1.5.  The  Participant  agrees  to  keep  details  regarding  the  PT/ILC  in 
confidence.  It  is  inappropriate  to  share  information  (specifically 
measurement  results)  regarding  the  artifacts  provided  in  the  proficiency 
test  final  report  or  to  seek  to  obtain  measurement  information  for 
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artifacts  prior  to  participation  in  the  proficiency  test  (i.e.,  participants 
will  share  data  only  with  the  coordinator,  analyst  or  NIST  staff). 

Exception:  Draft  En  values  or  notice  of  significant  error/bias  may  be 
communicated  to  individual  laboratories,  with  follow-up  monitoring  by  the  ILC 
coordinator  or  by  NIST  to  enable  immediate  investigation,  corrective  action, 
and  retesting  (if  needed). 

1.6. 1.6.  The  participating  organization  will  regularly  attend  RMAP  meetings 
and  participate  in  PT/ILC  discussions. 

1.6. 1.7.  Participants  and  participating  organizations  will  be  technically 
qualified  for  the  measurement  parameter  of  interest. 

1.6.2.  NIST  WMD  reserves  the  right  to  refuse  participation  to  any 

organization,  even  if  they  are  technically  qualified,  if  the  participant  or 
participating  organization  has  violated  quality  policies,  procedures  for  care 
and  handling  of  artifacts,  repeated  technical  delays,  or  other  technical 
issues  that  could  potentially  adversely  affect  the  results  for  the  other 
participants.  Prior  to  WMD  issuing  a notice  of  refusal  to  a laboratory,  due 
efforts  will  be  made  to  notify  participants  in  this  category,  provide 
guidance,  and  ensure  that  the  laboratory  has  an  opportunity  for  corrective 
action. 

1.7.  Document  Control 

1.7.1.  Document  control  is  maintained  by  NIST  WMD,  with  approved 
documents  for  this  quality  system  published  and  available  for  unlimited 
distribution  on  the  World  Wide  Web  at  http://wwvv.nist.gov/labmetrology 
(select  Proficiency  Testing  Program).  Any  paper  copies  of  quality 
documents  used  for  any  purpose  must  match  the  revision  and  date  of  issue 
as  the  associated  document  posted  at  the  website.  WMD  will  make 
general  announcements  to  all  staff  and  delegates  when  updates  have  been 
made.  The  approved,  controlled  copies  of  publications  will  be  electronic 
and  retained  on  the  website.  Copies  will  be  retained  for  local  use  and  back- 
up on  NIST  or  WMD  networks. 

1.7.2.  All  documentation  related  to  the  scope  of  this  quality  manual  is 
reviewed  and  approved  for  use  by  the  Technical  Manager  prior  to  issue. 
The  master  list  of  approved  documents  is  located  at 
http://wvvw.nist.gov/labmetrology. 
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1.7.3.  All  documentation  will  be  technically  reviewed  for  content  and 
policy  and  assigned  a publication  number  as  per  the  requirements  of 
NIST’s  editorial  review  and  approval  process,  the  Washington  Editorial 
Review  Board  (WERB)  in  Gaithersburg,  MD 

1.7.4.  Documentation  related  to  this  quality  system  is  reviewed  during 
Internal  Audits  and  through  regular  use  by  all  participants  to  ensure 
continuing  suitability.  Documents  are  also  reviewed  by  personnel  during 
each  use.  Any  issues  of  suitability  are  communicated  to  the  Quality 
Manager  in  writing. 

1.7.5.  During  the  document  revision  process,  any  document  that  has  been 
revised  will  be  removed  from  the  website  and  replaced  with  the  most 
current  revision.  Historical  documents  may  be  maintained  on  the  website 
if  they  have  been  suitably  marked  as  ‘"OBSOLETE,  REFERENCE  ONLY” 
so  that  the  historical  nature  is  obvious  if  used. 

1 .7.6.  Any  invalid  or  obsolete  documents  maintained  by  NIST  WMD  for 
historical  purposes  are  marked  as  “OBSOLETE,  REFERENCE  ONLY.” 

1.7.7.  Any  revisions  to  documents  will  be  reviewed  and  approved  by  the 
Technical  Manager.  Minor  changes  to  documentation  will  be  described  in 
the  revision  history  for  the  document.  Major  changes  to  the  documents 
will  be  identified  by  the  word  processing  software  “track  changes” 
function  and  maintained  by  the  Technical  Manager.  Hand-written  changes 
will  not  be  used. 

1.8.  Request,  Tender  or  Contract  Review 

1.8.1.  NIST  WMD  program  is  not  a commercial  PT/ILC  organization. 

- All  PT/ILC  activities  occur  as  a result  of  WMD  direction  and/or  RMAP 
participation  in  the  development  of  PT/ILC.  The  contract  review  function 
is  completed  as  a result  of  all  members  reaching  consensus  in  the 
development  of  the  PT/ILC  as  documented  on  the  PT/ILC  Planning 
Checklist  (Form  1)  for  the  RMAP's  form.  All  participants  must  agree  to 
abide  by  all  elements  of  POLICY  FOR  PARTICIPATION . Any  individual 
request  for  a new  member  or  a member  of  another  RMAP  to  participate  in 
a PT/ILC  that  has  already  been  developed  and  is  operating  must  be 
submitted  to  the  NIST  WMD  office  for  review,  documentation  and 
approval  by  the  Technical  Manager.  Records  of  the  PT/ILC  Planning 
Checklist,  individual  requests  to  participate,  and  any  associated  changes 
are  reviewed,  approved,  and  maintained  by  the  NIST  WMD  office. 
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1.8.2.  PT/ILCs  are  communicated  to  participants  outside  of  the  RMAPs, 
usually  via  email  or  the  NIST  WMD  website.  The  NIST  WMD  website 
provides  an  informational  overview  for  each  PT/ILC  and  whether  or  not  it 
has  openings  for  individual  requests  to  participate.  Special  requirements 
for  participation  are  noted/posted  where  applicable. 

1.8.3.  Any  changes  in  the  scheme  design  or  operation  of  a PT/ILC  are 
communicated  to  all  participants  in  writing.  Changes  are  maintained  with 
WMD  records  for  each  PT/ILC. 

1.8.4.  When  PT/ILC  functions  are  coordinated  with  or  on  behalf  of 
NVLAP,  specific  changes  to  the  operation  of  a PT/ILC  that  are  in  conflict 
with  this  quality  manual  must  be  documented  and  retained  on  file. 

1.9.  Procurement  of  Services  and  Supplies 

1.9.1.  NIST  WMD  uses  the  Federal  Acquisition  Requirement  (FAR) 
policies  and  procedures  for  procurement  of  all  services  and  supplies.  The 
Technical  Manager  or  Designee  develops  and  documents  technical 
specifications,  tolerances,  and  other  requirements  for  all  procured  services 
and  supplies,  which  are  used  in  selecting  suppliers.  Upon  receipt  of  items 
that  affect  the  quality  of  PT/ILC  services  provided,  the  items  are  inspected 
for  conformance  to  the  documented  technical  requirements.  For  certain 
artifacts  the  reference  laboratory  may  be  required  to  perform  the  final 
inspection  in  order  to  ensure  that  the  artifact  is  suitable  for  use  in  the 
PT/ILC.  Final  reports  and  analyses  are  reviewed  and  documented  by  the 
Technical  Manager  or  Designee.  Records  of  procurement  and  inspection 
are  maintained  by  NIST  WMD. 

1.9.2.  When  items  are  used  for  PT/ILC  activities  that  do  not  belong  to 
WMD,  suitable  evaluation  and  assessment  must  be  conducted  and 
documented  to  ensure  that  artifacts  are  of  sufficient  quality  to  meet  the 
stated  objectives  of  the  PT/ILC. 

1.10.  Client  Feedback 

1.10.1.  Feedback  from  participants  is  sought  during  each  RMAP  meeting, 
in  order  to  improve  PT/ILCs  that  are  developed  within  the  scope  of  this 
document.  The  PT/ILC  Feedback  and  Complaints  Form  (Form  4)  and  the 
WMD  Quality  System  Management  Review  Form  (Form  5)  will  be  used  to 
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ensure  the  uniform  collection  and  consolidation  of  feedback  from  all 
RMAPs. 

1 .10.2.  Complaints  and  feedback  may  be  recorded  by  an  ILC  Coordinator, 
a delegate  of  the  ILC,  or  the  WMD  office  staff.  Records  of  such  feedback 
or  complaints  and  their  associated  investigation  and  corrective  actions  are 
maintained  through  completion  of  the  PT/ILC  Feedback  and  Complaint 
Form  (Form  4)  or  the  WMD  Nonconformance/Corrective 
Action/Preventive  Action  (NCAP)  Form  (Form  7).  Evaluation  of  criteria 
for  conformance  and  non-conformance  is  performed  by  the  Technical 
Manager  in  accordance  with  Section  1.11. 

1.11.  Control  of  Nonconforming  Activities  and  Corrective  Action 

1.11.1.  In  the  event  that  any  aspect  of  activities  within  the  scope  of  this 
document  is  identified  as  nonconformity  to  the  procedures  or  the  stated 
requirements,  it  is  the  responsibility  of  the  Technical  Manager  to 
investigate  and  determine  appropriate  corrective  action.  The  corrective 
action  process  is  applied  to  all  identified  nonconformities  and  shall  be 
appropriate  to  the  magnitude  of  the  problem  encountered. 

1 .1 1 .2.  All  aspects  of  the  nonconformance  are  recorded  on  the  WMD 
Form  7 (NCAP).  It  is  the  responsibility  of  the  Technical  Manager 
to: 

1 . 1 1 .2. 1 . investigate  and  determine  root  cause  of  nonconformance; 

1.11 .2.2.  evaluate  the  significance  of  the  nonconformance; 

1 . 1 1 .2.3.  identify  potential  corrective  actions  and  select  the  action  most 
likely  to  eliminate  the  problem  and  prevent  it  from  reoccurring; 

1 . 1 1 .2.4.  ensure  work  is  halted  if  necessary; 

1 . 1 1 .2.5.  assign  a timescale  for  completion  of  corrective  action; 

1.1 1.2.6.  obtain  funding  if  needed;  and 

1 . 1 1 .2.7.  initiate  corrective  actions  as  required. 

1.11.3.  If  necessary,  participants  are  notified  in  writing  and 
nonconforming  materials  or  reports  are  recalled.  Upon  completion  of 
corrective  actions,  the  Technical  Manager  may  authorize  the  resumption  of 
the  process/scheme. 

1.11.4.  Any  corrective  actions  are  monitored  by  the  Quality  Manager 
during  the  Internal  Audits  to  ensure  that  the  corrective  action  was 
effective. 
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1.12.  Preventive  Action 

1.12.1.  All  operational  procedures  are  reviewed  during  the  Internal  Audit 
and  upon  each  use  by  all  staff  and  delegates  in  order  to  identify 
opportunities  for  improvement.  Any  preventive  actions  and  their 
associated  action  plans  are  identified  and  recorded  on  WMD  NCAP  Form 
(Form  7).  Results  of  preventive  actions  are  monitored  and  reviewed 
during  Internal  Audits  and  Management  Reviews  in  order  to  understand 
the  effectiveness  of  the  preventive  action. 

1.13.  Records 


1.13.1.  The  following  types  of  records  are  identified  and  indexed  as 
quality  records  for  NIST  WMD  PT/ILC  activities.  They  are  collected  at 
time  of  generation,  and  stored  by  marked  section  in  file  cabinets  or  in 
electronic  version  on  computer.  The  retention  times  for  all  documents  are 
at  least  ten  years  after  the  completion  of  the  test  or  retirement  (or  loss)  of 
the  artifact  or  proficiency  test.  After  ten  years,  the  Quality  Manager  may 
dispose  of  records  at  his/her  discretion. 

1.13.1.1.  Master  Document  List 

1 . 1 3. 1 .2.  Previous  editions  of  controlled  documents  and  obsolete  documents 

1.13.1.3.  Client  Feedback 

1 . 1 3. 1 .4.  Artifact  History  Records  and  Inventory  List 

1.13.1.5.  Customer  Complaint/Feedback  Log  and  Information 

1.13.1.6.  Nonconformities 

1.13.1.7.  Corrective  Actions 

1.13.1.8.  Preventive  Actions 

1 . 1 3. 1 .9.  Proficiency  Test/Interlaboratory  Comparison  Records 

1.13.1.10.  Proficiency  T est/Interlaboratory  Comparison  Reports 

1.13.1.11.  Internal  and  External  Audits 

1.13.1.12.  Management  Reviews 

1.13.1.13.  Training  Schedule  and  Training  Records  (i.e.,  Form  3) 

1.13.1.14.  Assignment  of  Deputy  Technical  and  Quality  Manager 

1.13.1.15.  Software  Validation  Records 

1.13.1.16.  Correspondence  Records 


1.13.2.  The  storage  facilities  for  records  are  at  the  NIST  WMD  office. 

Back-up  copies  of  electronic  records  are  maintained  on  the  NIST  network. 
NIST's  Information  Technology  (IT)  back  up  services  and  security  features 
are  administered  following  policies  and  procedures  set  by  the  NIST  Office 
of  Chief  Information  Officer. 
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1.13.3.  The  records  are  stored  in  the  NIST  WMD  offices  so  that  only 
employees  have  access  to  the  hard  copy  records.  Electronic  copies  of 
records  are  stored  on  the  hard  drives  and  network  of  NIST  WMD 
computers  and  are  appropriately  password  protected. 

1.13.4.  To  prevent  unauthorized  amendment  of  data,  Electronic  Data  is 
protected  via  password  protection  of  operational  features  for  each  person’s 
PC,  and  backed  up  to  CD  ROM  media,  to  prevent  unauthorized 
amendment  of  data. 

1.13.5.  Technical  records  contain  information  as  prescribed  in 
REPORTING  and  are  maintained  as  a quality  record. 

1.14.  Internal  & External  Audits 

1.14.1.  An  Internal  Audit  of  the  quality  system  related  to  this  document  is 
completed  annually,  as  scheduled  by  the  Quality  Manager.  The  audit  may 
be  performed  by  sections  so  that  the  entire  quality  system  is  reviewed 
annually. 

1.14.2.  The  Quality  Manager  is  primarily  responsible  for  the  planning  and 
conduct  of  the  Internal  Audit.  The  schedule  for  the  Internal  Audit  will  be 
determined  at  the  time  of  the  Management  Review  meeting.  The  Quality 
Manager  may  designate  individuals  to  perform  parts  of  the  Internal  Audit 
and  will  ensure  the  training  and  qualifications  of  such  individuals.  Internal 
Audit  Results  are  collected  and  summarized  on  the  WMD  Internal  Audit 
Form  (Form  6). 

1.14.3.  The  Internal  Audit  shall  review  all  elements  and  activities 
associated  within  the  scope  of  this  document.  During  the  Internal  Audit, 
all  documents  and  program  elements  are  reviewed  for  continuing 
effectiveness  and  potential  for  improvement.  Any  nonconformities 
identified  throughout  the  previous  year  are  reviewed  for  closure  and 
effectiveness.  Any  nonconformities  identified  during  the  Internal  Audit 
will  be  addressed  by  following  the  policy  in  ADMINISTRATION  - 
Control  of  Nonconforming  Activities  and  Corrective  Action. 

1.14.4.  Any  External  Audits  that  are  authorized  and  conducted  will  follow 
appropriate  or  applicable  standards  as  determined  by  the  Technical  and/or 
Quality  Manager.  Such  records  will  be  retained  with  other  auditing 
records  and  discussed  during  Management  Reviews. 
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1.15.  Management  Review 

1.15.1.  RMAP  PT/ILC  Quality  System  Review 


1.15.1.1.  NIST  WMD  conducts  management  reviews  of  the  quality  system 
and  PT/ILC  activities  as  applicable  within  each  RMAP  during  the 
annually  scheduled  meeting  of  the  RMAP.  Any  suggestions  for 
changes  or  improvements  are  summarized  in  report  form  (Forms  4 and 
5)  for  discussion  at  the  annual  Management  Review  meeting. 


1.15.2.  NIST  WMD  PT/ILC  Quality  System  Review 


1.15.2.1.  NIST  WMD  conducts  a Management  Review  meeting  of  the 
quality  management  system  annually,  as  scheduled  by  the 
Technical  Manager.  The  Management  Review  meeting  discusses: 

1 . 1 5.2. 1 . 1 . Suitability  and  effectiveness  of  the  Quality  System 

1.15.2.1.2.  The  Quality  and  applicability  of  PT/ILC  Activities 

1 .15.2.1 .3.  Reports  from  RMAP  meetings  (Forms  4 and  5) 

1 . 1 5.2.1 .4.  Reports  from  Internal  Audits  (Form  6) 

1.15.2.1.5.  Reports  of  External  Audits  (e.g.,  NVLAP,  A2LA,  L.A.B., 
other  requested  body) 

1.15.2.1.6.  Client  feedback/complaints 

1 .15.2.1 .7.  Resources  (facilities,  standards,  and  equipment) 

1.15.2.1.8.  Outstanding  items  from  previous  Management  Review 

1.15.2.1.9.  Staff  Training 

1.15.2.1.10.  Other  relevant  reports 


1.15.2.2.  The  minutes  of  the  Management  Review  meeting  are  recorded  and 
any  actions  or  improvement  suggestions  are  assigned  to  appropriate 
personnel  with  an  estimated  completion  date  to  ensure  completion 
within  an  appropriate  and  agreed  upon  timescale. 


1.16.  Management  Staffing  and  Training 


1.16.1.  Training 

1.16.1.1.  The  Technical  Manager  has  responsibility  for  the  management  of 
all  PT/ILC  activities  associated  with  NIST  WMD.  The  Technical 
Manager  is  responsible  for  ensuring  that  all  personnel  who  have  any 
responsibilities  associated  with  any  PT/ILC  activities  are  sufficiently 
trained. 
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1.16.1.2.  The  Coordinator,  Delegate,  and  Analyst  positions  as  assigned  for 
PT/ILC  activities  must  have  adequate  training  to  ensure  that  they  have 
the  technical  capability  to  complete  the  duties  as  assigned.  It  is  the 
responsibility  of  the  Technical  Manager  to  annually  develop  a training 
plan  and  ensure  that  essential  training  is  conducted  to  ensure  the 
competency  of  all  personnel  involved  in  the  PT/ILC  activities. 

1.16.1.3.  Participant  training  and  qualifications  shall  be  documented  using 
the  PT/ILC  Training  and  Qualifications  form  (Form  3)  to  verify 
adequate  training  and  ensure  that  they  possess  the  technical  capability 
to  participate  within  specific  PT/ILC  parameters. 

1.16.1.4.  Records  of  training  for  all  personnel  involved  with  PT/ILC 
management  activities  are  maintained  using  Form  3 and  database 
developed  and  maintained  by  NIST  WMD  for  training  records. 

1.16.2.  Staffing 

1.16.2.1.  The  Technical  Manager  is  responsible  for  ensuring  sufficient 
staffing  for  each  PT/ILC  and  that  all  personnel  involved  in  PT/ILC  are 
qualified.  The  Technical  Manager  reviews  the  training  records  of  all 
personnel  involved  to  ensure  the  effective  management  of  all  PT/ILC 
activities.  Records  of  training  for  all  personnel  involved  with  PT/ILC 
management  activities  are  maintained  using  a database  developed  and 
maintained  by  NIST  WMD  for  training  records. 

1.17.  Collusion  and  Falsification  of  Results 

1.17.1.  Each  PT/ILC  activity  is  designed  to  ensure,  as  is  practicable,  that 
the  possibility  for  collusion  and  falsification  of  results  is  minimized.  It  is 
the  responsibility  of  the  PT/ILC  Coordinator  to  monitor  the  activities  of 
the  PT/ILC  and  to  ensure  that  there  is  no  communication  of  the  assigned 
values  to  or  among  participants  until  the  results  of  the  PT/ILC  activity 
have  been  collated  in  a draft  report.  In  addition  to  the  PT/ILC  coordinators 
monitoring  activities  to  this  end,  each  participant  in  PT/ILC  activities 
within  the  scope  of  this  document  is  required  to  comply  with  all  elements 
of  POLICY  FOR  PARTICIPATION. 

1.18.  Data  Processing  Equipment 

1.18.1.  NIST  WMD  utilizes  personal  computers,  laptop  computers,  and 
scientific  calculators  for  data  entry  and  statistical  analysis  of  PT/ILC 
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activities,  all  of  which  are  adequate  for  the  assigned  task.  Deficiencies 
noted  with  data  processing  equipment  are  brought  to  the  attention  of  the 
Technical  Manager,  and  are  addressed  during  Internal  Audits  and 
Management  Reviews  as  appropriate. 

1.18.2.  The  IT  activities  of  NIST  or  the  IT  Department  of  Delegate 
organizations  are  responsible  for  the  effective  operation  of  laptop  and 
personal  computers.  Calculators  are  a disposable  item  and  are  replaced  by 
the  user  if  any  abnormal  operation  is  detected.  Use  of  appropriate 
significant  digits  is  considered  in  data  evaluation  and  addressed  as  needed 
during  ongoing  training  of  staffdelegates. 

1.18.3.  Commercial  Software  programs  such  as  Microsoft  Excel  do  not 
require  further  validation.  Sub-programs  such  as  worksheets  for  the 
analysis,  evaluation,  and  reporting  of  results  for  activities  within  the  Scope 
of  this  document  require  validation  before  use  as  per  the  PT/ILC  Software 
Validation  Procedure  (Procedure  1). 

1.18.4.  Records  of  objective  evidence  for  the  validation  and  revision  date 
of  the  software  are  maintained  at  NIST  WMD  by  the  Quality  Manager. 
The  distribution  of  any  such  software  is  accomplished  through  the  NIST 
WMD  website,  by  email,  or  by  sharing  of  files  through  other  electronic 
means.  The  software  on  the  NIST  WMD  website  is  the  master  copy  of  the 
software  and  it  is  backed  up  routinely  by  NIST  IT  procedures. 
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2.  PLANNING 

The  general  PT/ILC  requirements  are  plans  developed  in  accordance  with  NISTIR  7082, 
during  Management  Review  and  RMAP  meetings. 

A PT/ILC  begins  with  the  planning  activities  performed  by  an  advisory  group.  This 
group  may  be  assembled  by  the  Technical  Manager  for  national  level  PT/ILC  activities  or 
may  consist  of  an  RMAP  group  (or  subgroup).  The  advisory  group  plans  the  PT/ILC 
activity  by  completing  the  PT/ILC  Planning  Checklist  Form  (Form  1). 

2.1.  Technical  Protocol  Development 

2,1.1.  The  Technical  Manager  or  Delegate  develops  the  Technical 

Protocol  for  the  PT/ILC  based  upon  the  information  obtained  in  the 
associated  PT/ILC  Planning  Checklist  form  (Form  1).  The  Technical 
Protocol  shall  contain  the  following  information,  as  relevant: 

• Environmental  Conditions  for  Realizing  the  Measurement  Results  for  the 
Artifact; 

• Environmental  Conditions  for  Storage  and  Transport  for  the  Artifact; 

• Preparatory  Information  for  the  Artifact(s); 

• Packaging  and  Shipping  Information  of  the  Artifact  and  tracking 
requirements; 

• Communication  to  the  participants  to  treat  the  artifact  in  the  same  manner 
as  routine  measurements  for  the  method  prescribed  or  other  specified 
instructions  on  handling; 

• Instructions  of  the  measurement  method  to  be  used  in  the  PT/ILC; 

• Reporting  Requirements  for  Participating  Laboratories: 
o Units  of  measurement; 

o Number  of  significant  digits; 
o Date  of  measurement; 

o Reporting  reference/basis  of  measurement  (e.g.,  environmental 
conditions,  reference  temperature); 

o Measurement  value  and  its  associated  expanded  uncertainty  (the 
reported  measurement  uncertainty  must  be  the  actual  calculated 
uncertainty,  not  just  a transfer  ratio); 
o Uncertainty  analysis  (components  included  or  not  included);  and 
o The  laboratory  report  on  the  method  used  if  different  than  prescribed 
in  the  Technical  Protocol;  and 

• Directions  on  how  to  return  or  submit  the  data. 
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3.  IMPLEMENTATION 

3.1.  Preparation  of  the  Artifact 

3.1.1.  The  artifact(s)  is/are  selected  (as  described  in  Section  1.9)  for  the 
PT/ILC  based  upon  requirements  documented  in  the  PT/ILC  Planning 
Checklist  and  Procurement  of  Services  and  Supplies.  The  artifact(s)  may 
be  loaned  from  organizations  such  as  NIST  WMD  or  other  laboratories  as 
long  as  appropriate  planning  is  taken  to  ensure  that  the  reference  value  has 
not  been  previously  known  by  participating  laboratories. 

3.1.2.  The  Reference  Laboratory  (where  used)  and  Pivot  Laboratory 
(where  used)  are  qualified  based  upon  information  obtained  in  the  PT/ILC 
Planning  Checklist.  Records  of  qualification  for  the  Reference  Laboratory 
and  Pivot  Laboratory  are  created  per  Collaboration  and  Subcontracting 
(Section  1.4). 

3.1.3.  The  Technical  Manager  will  develop  any  additional  processes  to 
determine  the  stability  of  the  artifact  as  necessary.  The  artifact(s)  is/are 
then  sent  to  the  Reference  Laboratory  for  the  establishment  of  its  reference 
value(s)  or  to  the  Coordinating  Laboratory  where  the  reference  value  is 
selected  using  alternative  methods. 

3.1.4.  The  Technical  Manager  or  delegate  reviews  qualifications  for 
potential  participating  laboratories  via  training  databases  maintained  by 
NIST  WMD  and/or  completed  on  the  PT/ILC  Planning  Checklist 
(Form  1). 

3.1.5.  The  PT/ILC  Coordinator  then  develops  a schedule  for  the  PT/ILC 
activity  based  upon  the  final  list  of  participating  laboratories  and  the 
number  of  pivot  measurements  (when  used).  The  schedule  may  be 
communicated  and  tracked  in  the  most  convenient  and  practical  way  for 
the  PT/ILC  Coordinator. 

3.1.6.  The  PT/ILC  is  initiated  according  to  the  schedule. 

3.2.  Packaging/Shipping/Handling 

3.2.1.  The  PT/ILC  Coordinator  is  responsible  for  shipping  the  artifact(s) 

using  appropriate  packaging  methods  for  the  particular  artifact(s),  along 
with  any  additional  considerations  that  were  identified  in  the  PT/ILC 
planning  checklist  Planning  Checklist  (Form  1). 
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3.2.2.  The  PT/ILC  Coordinator  and  each  Participating  Laboratory  is 

responsible  for  ensuring  that: 

• Standards  are  appropriately  packaged; 

• Shipping  containers  are  appropriately  labeled;  PT/ILC  instructions  are 
securely  attached  to  the  product  packaging  of  individual  units  and  designed 
to  remain  legible  and  intact  for  the  period  of  use  in  the  PT/ILC; 

• The  PT/ILC  instructions  and  forms  should  generally  be  included  in  the 
shipping  container.  A minimum  of  a packing  list  and  the  WMD  shipping 
address  will  be  included  with  all  PT/ILC  artifacts. 

• Standards  are  shipped  by  an  appropriate  commercial  carrier  or  by  other 
methods  as  identified  in  the  ILC  Schedule  (Form  8)  & shipping  instructions; 
and 

• Standards  are  protected  from  adverse  environmental  conditions  (i.e., 
protection  of  heat,  cold,  x-rays)  as  appropriate. 

• Any  damage  or  loss  sustained  by  any  standards,  artifacts,  or  shipping 
containers,  is  immediately  reported  to  NIST/WMD. 

3.3.  Materials  Handling  and  Storage 

3.3.1.  The  PT/ILC  Coordinator  is  responsible  for  short-term  storage  of 
artifacts  during  gaps  in  the  schedule,  and  the  Technical  Manager  is 
responsible  for  long-term  storage  of  artifacts.  Whether  artifacts  are  in  long 
term  or  short  term  storage,  the  responsible  party  is  accountable  for 
identification,  preservation,  and  segregation  of  all  artifacts  from 
incompatible  environments  or  materials  such  as  high  humidity, 
uncontrolled  temperatures,  or  chemicals  and  other  items  so  that  the 
artifacts  do  not  deteriorate  during  storage. 

3.3.2.  The  Technical  Manager  maintains  the  inventory  list  of  artifacts  in 
long-term  storage  and  is  responsible  for  developing  schedules  of  periodic 
inspection  for  possible  deterioration  as  appropriate  for  the  artifacts. 


NIST  WMD  Quality  Manual  for 

Proficiency  Testing  and  Interlaboratory 
Comparisons 

Section 

4.0 

Prepared  By: 

JCG 

Date:  11/07/04 

Subject:  Analysis 

EFF.  DATE 
01/01/05 

REV.  PAGE 

A 27  of  53 

4.  ANALYSIS 

4.1.  Determination  of  Assigned  Reference  Values  and  its  Associated  Uncertainty 

4.1.1.  The  method  for  determining  the  assigned  value  is  documented  on 
the  PT/ILC  Planning  Checklist  (Form  1)  prior  to  beginning  the  PT/ILC 
activity.  The  method  selected  must  be  based  upon  sound  metrological  and 
statistical  principles.  Documents  listed  in  the  REFERENCES  section  of 
this  manual  may  be  utilized,  but  this  list  is  not  necessarily  an  all-inclusive 
list.  Any  determination  of  the  reference  value  must  include  consideration 
of  stability  of  the  artifact  as  each  participant  performs  measurements.  The 
Expanded  Uncertainty  of  the  PT/ILC  will  be  the  combination  of  the 
reference  value  uncertainty  and  the  uncertainty  associated  with  stability. 
Some  examples  used  for  determining  reference  values  may  be: 

• The  Reference  Laboratory  (such  as  NIST  or  an  accredited  laboratory) 
measured  value  (stability  may  be  determined  by  Pivot  Laboratory 
measurements); 

• The  mean  of  the  Reference  Laboratory  measured  values  from  the 
beginning  and  end  of  the  PT/ILC; 

• The  median  of  the  results  from  accredited  laboratories; 

• The  median  of  one  result  from  each  participating  laboratory;  or 

• The  adjusted  median  of  the  participating  laboratories 

Documented  procedures  for  selecting  reference  values  will  be  used. 

4.2.  Performance  Evaluation  of  Participants 

4.2.1.  The  method  of  evaluating  performance  for  the  participants  and 
_ acceptable  limits  of  performance  is  documented  on  the  PT/ILC  Planning 

Checklist  (Form  1)  prior  to  beginning  the  PT/ILC  activity.  The  method 
selected  must  be  based  upon  sound  metrological  and  statistical  principles. 
Documents  used  in  the  REFERENCES  section  of  this  document  manual 
may  be  utilized,  but  this  list  is  not  necessarily  all-inclusive.  Some 
examples  of  determining  performance  of  the  Participant  may  include: 

• Determination  of  Normalized  Error  value,  En;  (En  > 1 is  considered  a 
failure) 

■ While  an  En  value  > 1 is  considered  a failure-;  the  analyst  may 
recommend  additional  follow  up  for  an  E^  value  between  0.5  and 
1. 


NIST  WMD  Quality  Manual  for 
Proficiency  Testing  and  Interlaboratory 
Comparisons 

Section 

4.0 

Prepared  By: 

JCG 

Date:  11/07/04 

Subject:  Analysis 

EFF.  DATE 
01/01/05 

REV.  | PAGE 

A 1 28  of  53 

• Precision  Tests  (The  reported  uncertainty  must  be  less  than  specified 
limits  for  some  applications); 

• Z score  value; 

• Participant  measurement  is  within  the  expanded  uncertainty  of  the 
PT/ILC; 

• Participant  value  shows  a bias  even  though  their  reported  uncertainty  is 
large  enough  to  pass  the  En  analysis;  and 

• Participant  measurement  result(s)  and  associated  expanded  uncertainty  is 
within  the  expanded  uncertainty  of  the  PT/ILC. 

• Expanded  uncertainty  (variations  due  to  laboratory  capabilities,  bias, 
uniformity  of  methodology  employed,  etc.) 

4.2.2.  When  assessing  the  technical  competency  of  a participating 
laboratory,  appropriate  technical  specialists  and  statisticians  (or  technical 
approaches  and  statistical  tools)  are  used  as  deemed  appropriate  by  the 
Technical  Manager.  Each  PT/ILC  report  will  contain  both  individual  and 
summary  analyses. 

4.2.3.  Other  Performance  Analysis  issues  to  evaluate  may  be: 

• Overall  performance  against  prior  expectations,  taking  measurement 
uncertainties  into  account; 

• Variation  within  and  between  laboratories,  between  RMAP  results,  and 
comparisons  with  any  similar  previous  schemes  or  published  precision 
data;  and 

• Variation  between  methods,  procedures,  or  equipment  used  if  applicable; 

• Evaluation  of  a laboratory's  timeliness  in  conducting  and  reporting  their 
PT/ILC  results;  and 

• Evaluating  the  laboratory’s  calibration  report  for  compliance  with  ISO 
17025  (Section  5.10)  criteria. 

4.3.  Outlier  Detection  Techniques 

4.3.1.  The  NIST/Sematech  e-Handbook  of  Statistical  Methods  provides 

guidance  on  outlier  detection  or  other  special  analysis  techniques.  Results 
of  PT/ILC  activities  within  the  scope  of  this  document  are  not  discarded. 
Results  that  could  be  considered  outliers,  or  results  from  laboratories 
performing  measurements  outside  of  their  normal  scope  of  recognition, 
may  not  be  used  for  statistical  summaries,  but  the  information  is  included 
in  draft  and  final  reports.  Any  suspect  data  will  be  verified  by  the 
Technical  Manager  or  Designee  to  ensure  that  the  data  entry  and 
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transcriptions  were  performed  correctly  and  that  any  data  entry  errors  were 
identified,  documented,  and  corrected.  Data  entry  errors  by  participants 
are  identified  by  feedback  or  reports  issued  by  NIST  WMD  to  the 
participant  (or  within  the  report),  who  will  use  their  corrective  action 
process  for  root  cause  analysis  and  response.  The  Technical  Manager  or 
Designee  is  responsible  for  follow-up  of  issued  nonconformities. 

4.4.  Assessing  the  Stability  of  the  Artifact 

4.4.1.  The  method  for  determining  artifact  stability  is  documented  on  the 
PT/ILC  Planning  Checklist  (Form  1)  prior  to  beginning  the  PT/ILC 
activity.  The  method  selected  must  be  based  upon  sound  metrological  and 
statistical  principles.  Documents  described  in  the  REFERENCES  section 
of  this  document  manual  may  be  used,  but  this  list  is  not  necessarily  all- 
inclusive.  Limits  of  acceptable  stability  for  the  artifact  are  also  determined 
and  recorded  on  Form  1 prior  to  beginning  the  activity.  It  is  the 
responsibility  of  the  Technical  Manager  to  communicate  the  limits  of 
artifact  stability  to  the  Pivot  Laboratory  so  that  they  can  evaluate  the 
artifact  stability  during  the  PT/ILC  and  alert  the  Technical  Manager  if 
stability  limits  have  been  exceeded.  If  established  limits  have  been 
exceeded,  it  is  the  responsibility  of  the  Technical  Manager  to  determine 
how  to  proceed. 

4.4.2.  Some  examples  of  determining  stability  may  include: 

• The  measured  deviation  between  the  opening  and  closing  measurements 
performed  by  the  Reference  Laboratory; 

• The  standard  deviation  of  the  Pivot  Laboratory  measurements; 

• The  uncertainty  of  linear  regression  of  the  Pivot  Laboratory 
measurements;  and 

• The  standard  deviation  of  participant  measurements. 

4.5.  Assessing  Homogeneity7  of  Artifact 

4.5.1.  If  homogeneity  is  determined  to  be  applicable  for  the  PT/ILC 

activity,  the  assessment  of  homogeneity  is  performed  after  the  test  material 
has  been  packaged  in  its  final  form  and  before  distribution  to  participants 
unless,  for  example,  stability  studies  indicate  that  it  should  be  stored  in 
bulk  form.  In  some  cases,  an  intermediate  homogeneity  check  may  be 
necessary,  for  example,  before  sealing  into  appropriate  packaging. 
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4.5.2.  The  provider  or  its  collaborators,  where  appropriate,  shall  use  a 
statistically  random  selection  of  a representative  number  of  samples  from  a 
batch  of  test  material  to  assess  the  homogeneity  of  the  material.  This 
assessment  procedure  shall  be  documented  and  conducted  in  accordance 
with  acceptable  statistical  designs,  for  example,  analysis  of  variance  on 
replicate  results  under  repeatability  conditions.  In  the  case  of 
measurement  artifacts,  preliminary  stability  checks  shall  be  made  and 
periodic  checks  of  assigned  property  values  should  be  carried  out 
throughout  the  course  of  the  scheme. 

4.5.3.  If  appropriate,  limits  of  acceptable  homogeneity  for  the  artifact  are 
also  determined  and  recorded  on  the  PT/ILC  Planning  Checklist  (Form  1) 
prior  to  beginning  the  activity. 

4.5.4.  It  is  the  responsibility  of  the  Technical  Manager  to  communicate 
the  limits  of  artifact  homogeneity  to  the  Pivot  Laboratory  so  that  they  can 
evaluate  the  artifact  homogeneity  during  the  PT/ILC  and  alert  the 
Technical  Manager  if  the  established  limits  have  been  exceeded.  If 
established  limits  have  been  exceeded,  it  is  the  responsibility  of  the 
Technical  Manager  to  determine  how  to  proceed. 


REPORTING 


5.1.  Schedules 

5.1.1.  The  PT'ILC  Coordinator  develops  and  publishes  Schedule  & 
Shipping  instructions  based  upon  the  information  gathered  in  the  PT/ILC 
Planning  Checklist  (Form  1).  The  schedule  will  be  sent  to  all  participants 
and  may  be  posted  at  the  NIST  WMD  website. 

5.2.  Preliminary  Reports 

5.2.1.  Preliminary’  or  Draft  Reports  may  be  issued  to  the  participant(s)  at 
the  discretion  of  the  Technical  Manager.  Since  such  reports  may  be 
requested  before  the  closing  measurement  of  the  reference  laboratory,  the 
Analyst  will  need  to  perform  evaluations  based  upon  a given  reference 
value  or  the  opening  measurement  on  the  artifact  by  the  Reference 
Laboratory  and  the  evaluation  of  stability  performed  thus  far  in  the  PT/ILC 
activity.  All  participants  are  to  review  their  preliminary'  report  to  ensure 
that  all  data  on  the  report  corresponds  to  the  data  that  was  submitted  by  the 
laboratory'. 

5.3.  Final  Reports 

5.3.1.  The  content  of  the  Final  Report  will  vary'  depending  on  the  purpose 
of  a particular  scheme,  but  each  report  shall  be  clear  and  comprehensive 
and  include  data  on  the  distribution  of  results  from  all  participants, 
together  with  an  indication  of  the  performance  of  individual  participants. 
Final  Reports  are  delivered  prior  to  or  at  the  RMAP  meeting  for  discussion 
and  comments. 

5.3.2.  Final  Reports  issued  for  PT/ILC  activities  within  the  scope  of  this 
document  shall  contain: 

• Elements  contained  in  the  PT/ILC  Reporting  Checklist  (Form  2); 

• .Any  other  suggestions,  recommendations  or  general  comments; 

• Conclusions; 

• Comments  on  the  performance  both  of  individual  participants  and  on 
participation  as  a whole;  and 

• Possible  sources  of  error  (with  reference  to  extreme  results)  and 
suggestions  for  improving  performance. 
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APPENDIX  A - FORMS 


Number 

Title 

1 

PT/ILC  Planning  Checklist 

2 

PT/ILC  Reporting  Checklist 

3 

PT/ILC  Training  & Qualifications 

4 

PT/ILC  Feedback  & Complaint 

5 

WMD  Quality  System  Management  Review 

6 

WMD  Internal  Audits 

7 

WMD  Nonconformance,  Corrective  Action,  Preventive  Action 
(NCAP) 

8 

Schedule  & Shipping  (nonmandatory) 
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Form  1:  PT/ILC  Planning  Checklist 

Page  1 of  3 


ILC  Region  & Planned  Reporting  Year: 

ILC  Title: 

ILC  Coordinator  Name: 

ILC  Coordinator  Address: 

Names,  position,  and  addresses  of  any 
other  personnel  involved  in  design  and 
operation  of  ILC: 

Form  Completed  by: 

Nature  and  Purpose  of  ILC  scheme: 


Identification  of  the  Measurand: 

Identification  and  description  of  Artifact(s): 

Description  of  nature  of  Artifact  and  why  it  was  chosen: 


Artifact  source  (owner): 


Describe  any  stability  issues  with  the  artifact: 


Identification  of  the  test  method  used  by  the  participants: 


Identification  of  the  type  of  equipment  required  by  the  participants: 


Identification  of  the  approximate  uncertainty  required  by  the  participants: 


Ranges  of  values  expected  from  the  participant  measurements: 
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Form  1:  PT/ILC  Planning  Checklist 
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Any  potential  difficulties  in  preparation  and  maintenance  of  homogeneous  materials  or 

the  provision  of  a stable  reference  value  for  the  artifact: 


Describe  any  homogeneity  issues,  as  applicable: 

Describe  any  environmental  issues  with  the  artifact  for  storage,  transport,  or 
measurement: 

Describe  any  special  preparation  issues  for  Artifact(s)  as  applicable: 


Packaging/Handling  issues,  Special  Containers  needed? 


The  Criteria  for  participation  in  the  ILC: 


Reference  Laboratory: 


Pivot  Laboratory: 


Identification  of  Expected  Participants: 
(identify  laboratory  and  personnel  to  be 
involved  in  measurement): 

Are  all  laboratories  and  personnel  qualified 
to  participate  (review  NIST  WMD  training 
databases) 

1 1 yes  O no  (if  no,  explain) 

Design  of  test  (i.e.,  petal,  loop,  etc.): 

* 

Describe  when  artifact  is  checked  and  how 
it  is  distributed: 
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Number  of  significant  figures  to  which 
results  are  to  be  reported: 

Precision  of  test: 

Smallest  differences  expected  between 
participants: 

Statistical  Method  for  determining  reference  value  of  artifact: 


Method  of  performance  evaluation: 


Acceptable  limits  of  stability  (from  pivot  measurements  and  overall): 


Acceptable  limits  of  homogeneity  (if  applicable): 
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PT/ILC  Identification: 

Coordinator: 

Date  of  Completion: 

Ensure  that  Final  Report  Contains  the  Following 


Title  Page  or  Introduction 

General  Description  and  title  of  the  ILC 

Names  and  affiliations  of  persons  involved  in  design  and  conduct 
of  PT/ILC 

1 1 Included 

Date  of  issue  of  the  report 

2 Included 

Report  number  and  clear  identification  of  the  PT/ILC 

1 1 Included 

Dates  (Time  period  of  conduct  of  test  from  beginning  to  end) 

1 1 Included 

RMAP  group 

[ ] Included 

Who 

. 

Reference  Laboratory 

~2  Included 

Pivot  Laboratory 

] Included 

Participants  (include  address  as  appropriate) 

1 1 Included 

What  and  Why 

Objectives 

2 Included 

Parameter,  Range,  Uncertainty  (e.g.,  mass,  1 kg,  best  calibration) 

2 Included 

Clear  description  of  the  items  or  materials  used,  including,  where 
appropriate,  details  of  sample  preparation  and  homogeneity 
testing 

1 1 Included 

Details  of  the  traceability  and  uncertainty  of  assigned  values 

1 1 Included 

How 

Summary  of  procedures  used  to  design  and  implement  the  scheme 
(which  may  include  reference  to  a scheme  protocol) 

1 1 Included 

Procedures  Used 

f]  Included 

Laboratory  Equipment  Identification  (e.g.,  which  balance,  which 
standards,  for  each  lab) 

1 1 Included 

Analysis 

-Av  ’ > . - -Ay 

Summary  of  procedure  used  to  establish  any  assigned  value 

2 Included 

Summary  of  procedures  used  to  statistically  analyze  the  data 

2 Included 

Software  title  and  version  used  in  data  analysis 

] Included 
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Tables  & Graphs 

Laboratory  test  results 

~~|  Included 

Statistical  data  and  summaries,  including  assigned  values  of 
Artifact(s),  and  range  of  acceptable  results 

| | Included 

Assigned  values  and  summary  statistics  for  test 
methods/procedures  used  by  other  participants  (if  different 
methods  are  used  by  different  participants) 

1 1 Included  N/A 

Tables  of  data,  precision  tests,  and  evaluation  data 

~|  Included 

Graphs  of  data,  standard  value/uncertainty  charges,  Y ouden  plots, 
misc.  graphs 

1 1 Included 

Conclusions 

Comments  on  participants'  performance  by  the  provider  and 
technical  advisors 

1 1 Included 

Summarization  of  Results 

~ ] Included 

Recommended  follow-up  and  corrective  action  (by  analyst/by 
NIST) 

1 1 Included 

Comments  on  any  technical  difficulties  raised  by  participants 

] Included  Q N/A 

Evaluation  of  the  responses  of  participants  performing  poorly 
(if  feedback  is  required) 

1 1 Included  Q N/A 

Advice,  where  appropriate,  on  the  interpretation  of  the  statistical 
analysis 

1 1 Included  Q N/A 

Return  to  NIST  WMD  For  Archival  - Copies  and/or  originals  of: 

Instructions 

_]  Included 

Schedules 

J Included 

Datasheets 

~~|  Included 

Formal  Calibration  Report  (where  requested) 

1 1 Included 

Submitted  Data 

Included 

Analysis  and  electronic  Documents/spreadsheets 

f]  Included 
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Name: 

Laboratory: 

Formal  Education 


SCHOOL  ATTENDED 

ATTENDED 

COURSES  COMPLETED 

CERTIFICATE 

AWARDED 

FROM 

TO 

YES 

NO 

Technical  Seminars  and  Workshops  Attended 


DATE  & PLACE  ATTENDED 

NAME  OF 

SEMINAR/WORKSHOP 

ATTENDED 

NUMBER  OF 
HOURS  OF 
INSTRUCTION 

CERTIFICATE 

RECEIVED 

YES 

NO 
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METROLOGY  DISCIPLINE 

TRAINING 

LEVEL 

YEARS  OF 
EXPERIENCE 

IN  THIS 
DISCIPLINE 

APPROVED  FOR 

PT/ILC  FUNCTIONS 

Mass 

Echelon  I 

Echelon  II 

Echelon  III 

Mass,  Fluid  Flow  (Volume) 

Echelon  I Gravimetric  - 
Large  Prover 

Echelon  II  Gravimetric  - 
Small  Volume  (Glassware) 

Echelon  II  Volumetric  - 
Large  Volume  Transfer 

Echelon  II  Volumetric  - 
Small  Volume  Transfer 

Length,  Dimensional,  Linear 

Length  Rulers 

Length  Tape 

Temperature 

Echelon  I 

Echelon  II 

Echelon  III 

Echelon  IV 

Echelon  V - 

Misc. 

Hydrometers 

Tuning  Forks 

Stop  Watches  and  Timers 

Hygrometer/Relative 

Humidity 

Grain  Moisture 

Reviewed  and  Approved  By: 

Date  of  Review: 

See  Training  Level  Codes  on  the  next  page. 
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TRAINING  LEVEL  CODE  LEGEND: 

BLANK:  No  experience  in  the  particular  field 

C:  Introductory  Experience  such  as  on-the-job  training  (OJT),  Computer  Based 
Instruction,  Informal  Instruction,  or  Formal  Instruction  that  is  over  ten  years  old. 

B:  Intermediate  Experience  such  as  formal  instruction  provided  in  house  or  by  a NIST 
WMD  approved  metrology  program. 

A:  Formal  Instruction  provided  by  a recognized  subject  matter  expert  such  as  original 
equipment  manufacturer  (OEM)  training,  or  NIST  seminars. 

APPROVED  FOR  PT/ILC  FUNCTIONS  LEGEND: 

The  qualifications  listed. below  are  in  ascending  order  of  ranking  (i.e.,  if  the  person  is 
qualified  as  a coordinator,  they  are  also  qualified  as  a participant,  but  are  not  necessarily 
qualified  as  an  analyst  or  for  reports.  A person  qualified  for  reports  is  qualified  for  all 
PT/ILC  activities  for  the  parameter. 

BLANK:  Not  approved  for  PT/ILC  activities  for  this  parameter. 

PARTICIPANT:  Approved  as  a participant  for  parameter. 

COORDINATOR:  Approved  as  a PT/ILC  coordinator  for  the  parameter. 

ANALYST:  Approved  as  PT/ILC  analyst  for  the  parameter. 

REPORTS:  Approved  to  develop  reports  under  the  supervision  of  the  Technical 
Manager  for  the  parameter. 
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Instructions:  Use  this  form  to  record  any  complaint  or  pertinent  communication 
received  about  NIST  WMD's  sendees  as  they  apply  to  the  conduct  of  PT/ILC  activities. 

If  the  complaint  or  comment  is  received  in  writing,  attach  the  complaint  to  this  record.  If 
the  complaint  or  comment  is  received  by  telephone,  fill  out  sections  4 and  5 completely. 

1 . Name  of  Person  completing  form 

3.  Check  one  of  the  following 

1 1 Complaint/communication  received  in 
writing  - copy  of  documentation  attached 

1 1 Complaint/communication  received  by 
telephone  - was  caller  requested  to  send 
documentation? 

2.  Date  of  receipt  of  complaint  or 
communication 

□ yes  □ no 

4.  If  the  complaint/communication  was  received  by  telephone,  provide  the  following 
information: 

Name  and  Address  of  person  making  complaint/communication 

Phone  number 

Fax  number 

5.  If  the  compliant/communication  was  received  by  telephone,  describe  the  issue: 

6.  Name  of  Responsible  Manager: 

7.  For  Quality  Manager  Use  Only 

Date  Logged: 

Complaint  Number: 

Date  C.A.R.  issued  (if  applicable): 

Date  Closed: 
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Date: 

Location: 

Personnel  Present: 

Agenda 

Record  of  Discussion,  and  Assigned  Actions,  and 
Estimated  Completion  Dates 

Suitability  and  Effectiveness  of 
Quality  System 

Quality  and  Applicability  of 

PT/ILC  Activities 

RMAP  Meeting  Reports 

Report  of  Internal  Audits 

Report  of  External  Audits 

Client  Feedback/Complaints 

Resources  (Facilities,  Standards, 
and  Equipment) 
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Outstanding  Items  from  Previous 
Management  Review 

Staff  Training 

Other  Reports 

Miscellaneous  Topics 
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PART  A 

Dates  of  Assessment: 

Lead  Assessor: 

Assessment  Team 

Members: 

Quality  System  Review  (all  sections  are  assessed  annually,  though  not  necessarily  at  the 
same  time) 

Section  1.1 

| | Assessed  | | Conforming  | | Nonconformities 

Section  1.2 

| | Assessed  [ | Conforming  | ] Nonconformities 

Section  1.3 

I~1  Assessed  [_]  Conforming  [_]  Nonconformities 

Section  1.4 

|~1  Assessed  [_]  Conforming  QJ  Nonconformities 

Section  1.5 

| | Assessed  | | Conforming  | | Nonconformities 

Section  1.6 

[_J  Assessed  |_)  Conforming  |_J  Nonconformities 

Section  1.7 

1 | Assessed  Q Conforming  Q Nonconformities 

Section  1.8 

1 1 Assessed  [_]  Conforming  | | Nonconformities 

Section  1.9 

1 1 Assessed  Q Conforming  Q]  Nonconformities 

Section  1.10 

FI  Assessed  | | Conforming  | | Nonconformities 

Section  1.11 

1 I Assessed  Q Conforming  Q Nonconformities 

Section  1.12 

1 1 Assessed  Q Conforming  Q Nonconformities 

Section  1.13 

1 | Assessed  Q Conforming  Q Nonconformities 

Section  1.14 

1 | Assessed  Q Conforming  Q Nonconformities 

Section  1.15 

1 | Assessed  Q Conforming  Q Nonconformities 

Section  1.16 

1 | Assessed  Q Conforming  | | Nonconformities 
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Section  1.17 

1 1 Assessed  Q Conforming  O Nonconformities 

Section  1.18 

| | Assessed  [_]  Conforming  | | Nonconformities 

PART  B 

PT/ILC  Quality  System  Review  (Select  a 10  % sample  (minimum  1 per  region)  of 

PT/ILCs  conducted  each  year  and  assess  all  elements  of  program) 

PT/ILC  Region  assessed: 

PT/ILC  activity  assessed: 

Section  2.1 

1 1 Conforming  Q Nonconformities  O Not  Developed  Yet 

Section  3.1 

1 1 Conforming  Q Nonconformities  Q Not  Developed  Yet 

Section  3.2 

1 1 Conforming  Q Nonconformities  Q Not  Developed  Yet 

Section  3.3 

1 1 Conforming  O Nonconformities  O Not  Developed  Yet 

Section  4.1 

1 1 Conforming  Q Nonconformities  Q Not  Developed  Yet 

Section  4.2 

1 1 Conforming  Q Nonconformities  Q Not  Developed  Yet 

Section  4.3 

1 1 Conforming  [f\  Nonconformities  Q Not  Developed  Yet 

Section  4.4 

1 1 Conforming  Q Nonconformities  O Not  Developed  Yet 

Section  4.5 

1 1 Conforming  Nonconformities  Q Not  Developed  Yet 

Section  5.1 

1 1 Conforming  Q Nonconformities  Q Not  Developed  Yet 

Section  5.2 

1 1 Conforming  Q Nonconformities  Q Not  Developed  Yet 

Section  5.3 

1 1 Conforming  Q Nonconformities  [f]  Not  Developed  Yet 
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I I Conforming  \ff\  Nonconformities  Q Not  Developed  Yet 


Section  5.4 


PAP 

IT  C 

Audit  for  Closure  of  Previous  Deficiencies 

| | Conforming  | | Nonconformities 

Attach  all  nonconformities  and  comments  using  the  NCAP  form  and  all  other  supporting 
documentation. 
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1.  Name  of  Person  completing  form: 


2.  Date: 


3.  Check  one  of  the  following: 
I I Nonconformity 
~]  Preventive  Action 


4.  Identification  of  Nonconformity  or  Preventive  Action: 


5.  Source  of  Discovery: 


6.  Investigation  Results: 


7.  Root  Cause: 


8.  Evaluation  of  Significance: 


9.  Action  Required?  I I Yes  n No 


10.  Does  PT/ILC  need  to  be  halted?  I I Yes  I I No 


1 1 . If  halted,  identify  location  of  Artifact/position  in  scheme; 


12.  Action  Taken: 


13.  Due  Date:  Participant  Notification  (in  writing) 


14.  Name  of  Responsible  Manager: 


15.  Date  PT/ILC  Resumed  (if  applicable): 
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PT/ILC  Identification: 

PT/ILC  Coordinator: 

Scheduled 

Date 

Laboratory 

Date 

Shipped  to: 

Shipper 

Tracking  # 

Date 

Received 

* As  long  as  the  coordinator  maintains  a tracking  system  for  the  artifacts  or  standards,  use 
of  a form  is  not  required. 


NIST  \VMD  Quality  Manual  for 
Proficiency  Testing  and  Interlaboratory 
Comparisons 

Section 

7.0 

Prepared  By: 

GLH 

Date:  3/28/2005 

Subject:  Appendix  B - Procedures  & 

Policies 

EFF.  DATE 
01/01/05 

REV.  PAGE 

A 50  of  53 

APPENDIX  B - PROCEDURES  & POLICIES 


Number 

Title 

1 

Software  Validation  Procedure 

2 

Selection  of  Delegates  Procedure  (To  be  developed). 

3 

NISTIR  7082  PT/ILC  Policy  & Plan 
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Procedure  1:  PT/ILC  Software  Validation  Procedure 

Background:  All  software  that  has  been  designed  to  perform  computations  associated 
with  reporting,  analyzing  or  evaluating  acceptance  criteria  for  PT/  ILC  activities  are 
required  to  be  validated  according  to  this  procedure.  Commercial  software  such  as 
Microsoft  Excel  or  MathCAD  does  not  require  validation,  but  worksheets  created  for 
PT/ILC  activities  do  require  validation. 

1)  Ensure  that  the  Software  contains  the  following  information: 

a.  Title  of  Software; 

b.  Date  of  Latest  Revision  (kept  within  the  electronic  file); 

c.  Identification  information  of  person  who  performed  the  last  revision. 

d.  Document  technical  specifications  used  (i.e.,  specific  formulas  for  air  or 
water  density,  coefficients  of  expansions). 

2)  The  Technical  Manager  or  Designee  performs  the  software  validation.  Under  no 
circumstance  is  the  author  of  the  software  to  be  evaluated  allowed  to  perform  the 
software  validation. 

3)  All  computational  functions  of  the  software  are  to  be  validated  after  each  revision 
(and  reviewed  at  least  annually).  Validation  is  accomplished  by  entering  a data 
set  into  the  software  to  be  validated,  and  comparing  the  results  obtained  to  results 
obtained  by: 

a.  Results  published  in  a textbook  or  other  appropriate  reference  for  the 
dataset; 

b.  Hand  or  calculator  computations  (which  are  documented); 

c.  A program  from  a different  software  platform; 

d.  A program  on  the  same  software  platform  that  is  significantly  different 
from  the  software  to  be  validated. 

4)  Document  the  validation  method  selected,  password(s),  and  supporting 
information  for  the  validation  process. 

5)  Sign  and  Date  the  documents  and  submit  them  to  the  Technical  Manager  for  final 
review  and  approval. 
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Procedure  2:  Selection  of  Delegates 


To  be  developed  at  a later  date. 
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Policy:  NISTIR  7082  "Proficiency  Test  Policy  and  Plan  (for  State  Weights  & 
Measures  Laboratories)"  (on-line  at  http://www. nist.gov/labmetrologv ) 


